With recent advances in generative AI, conversational models like ChatGPT have become feasible candidates for TAs. We investigate the practicality of using generative AI as TAs in introductory programming education by examining novice learners' interaction with TAs in a subgoal learning environment. To compare the learners' interaction and perception of the AI and human TAs, we conducted a between-subject study with 20 novice programming learners. Learners solve programming tasks by producing subgoals and subsolutions with the guidance of a TA. Our study shows that learners can solve tasks faster with comparable scores with AI TAs. Learners' perception of the AI TA is on par with that of human TAs in terms of speed and comprehensiveness of the replies and helpfulness, difficulty, and satisfaction of the conversation. Finally, we suggest guidelines to better design and utilize generative AI as TAs in programming education from the result of our chat log analysis.

There didn't seem to be anything strange about the new teaching assistant, Jill Watson, who messaged students about assignments and due dates in professor Ashok Goel's artificial intelligence class at the Georgia Institute of Technology. Her responses were brief but informative, and it wasn't until the semester ended that the students learned Jill wasn't actually a "she" at all, let alone a human being. Jill was a chatbot, built by Goel to help lighten the load on his eight other human TAs. "We thought that if an AI TA would automatically answer routine questions that typically have crisp answers, then the (human) teaching staff could engage the students on the more open-ended questions," Goel told Digital Trends. "It is only later that we became motivated by the goal of building human-like AI TAs so that the students cannot easily tell the difference between human and AI TAs. Now we are interested in building AI TAs that enhance student engagement, retention, performance, and learning."

There didn't seem to be anything strange about the new teaching assistant, Jill Watson, who messaged students about assignments and due dates in professor Ashok Goel's artificial intelligence class at the Georgia Institute of Technology. Her responses were brief but informative, and it wasn't until the semester ended that the students learned Jill wasn't actually a "she" at all, let alone a human being. Jill was a chatbot, built by Goel to help lighten the load on his eight other human TAs. "We thought that if an AI TA would automatically answer routine questions that typically have crisp answers, then the (human) teaching staff could engage the students on the more open-ended questions," Goel told Digital Trends. "It is only later that we became motivated by the goal of building human-like AI TAs so that the students cannot easily tell the difference between human and AI TAs. Now we are interested in building AI TAs that enhance student engagement, retention, performance, and learning."

As part of a three-week pilot study this quarter, half of the students in STATS 60: "Introduction to Statistical Methods" were assigned a RoboTA, an artificially intelligent robot teaching assistant (TA), to answer their questions by email. Students emailed different addresses based on the TA they were assigned. The emails were then funneled through a program, stripped of any identifying information and relayed to Lucas Janson, the TA for the class. Janson replied to all the emails without knowing whether they were intended for the human TA or the artificial intelligence (AI) TA. This study is double-blind, so neither the students nor the experimenters have information about the other, which helps prevent bias.

Ashok Goel, a professor at Georgia Tech, made the news this week with the revelation that one of the TA's that he used in his AI course was actually an "AI." Now, I have no reason to believe that Goel was trying to do something wrongheaded. I think he was just playing around. But the media love AI stories these days and have yet again led the public on a very wrong headed journey about what AI is and what it can be. Could a chatbot be an effective TA? It could certainly beat a rather disinterested TA.